Enabling Embedded Inference Engine with ARM Compute Library: A Case Study

If you need to enable deep learning on low-cost embedded SoCs, should you port an existing deep learning framework or should you build one from scratch? In this paper, we seek to answer this question by sharing our practical experience of building an embedded inference engine using the ARM Compute Library (ACL). The results show that, contradictory to conventional wisdom, for simple models, it takes much less development time to build an inference engine from scratch as opposed to porting existing frameworks. In addition, by utilizing ACL, we managed to build an inference engine that outperforms TensorFlow by 25%. Our conclusion is that, with embedded devices, we most likely will use very simple deep learning models for inference, and with well-developed building blocks such as ACL, it may yield better performance and result in lower development time if the engine is built from scratch.